Verification and Validation of Complex Systems

Verification and validation of complex system simulations are generally considered to be independent concepts. We consider the system model verified if there is a proof of the properties in question. We demonstrate that validation is a contravariant form of verification. We do this by considering the refutation tree produced by a resolution proof of the system properties. We indicate areas requiring further development. Keyword I: Complex systems Keyword II: Systems, modeling, theory, logic, theorem proving Introduction The goal of science is to explain the natural world. While some phenomena can be explained without reference to mathematics, most phenomena are modeled using mathematics (2). As our understanding increases, we can model more complex phenomena more completely . These complex models now most often require supercomputers to solve. These computational models — simulations — 1 Copyright  2002 by ASME August 30, 2002 11:19 add a new dimension of software and hardware correctness to the already complex question of scientific and mathematical correctness. The Defense Modeling and Simulation Office (DMSO) is a central repository of information on modeling, simulation, verification, and validation terminology. From DMSO’s site (www.dmso.mil) we find that verification is “The process of determining that a model or simulation implementation accurately represents the developer’s conceptual description and specification. Verification also evaluates the extent to which the model or simulation has been developed using sound and established software engineering techniques.” Validation is “ The process of determining the degree to which a model or simulation is an accurate representation of the real-world from the perspective of the intended uses of the model or simulation.” Validation, then, is the process of justifying (epistemology) a model to the physical processes it claims to represent (ontology). Note that the terms verification and validation are often treated as synonyms but in the specialized language of modeling and simulation they are not. Although mentioned together, the two are generally regarded as two distinct concepts. Our goal is to show that they can be related through the logical theory that establishes the system. Normal proofs proceed from the axioms to the conclusion in a covariant manner. Validation proceeds from the conclusions to the axioms in a contravariant manner. We use the covariantly generated proof tree to explore validation. Our approach is to generate the tree by resolution theorem proving. We assume the reader is familiar with standard logical terminology and a basic understanding of resolution theorem proving. Our approach is to generate a semantic structure based on resolution. Validation is a probabilistic statement on the acceptability of the tree. Before we can apply resolution theorem proving, we must first describe the logical system. We proceed as follows. Section is our logical foundations. Section reviews resolution theorem proving and establishes the verification aspect. Validation is considered in . Logical Systems in Carnap-Hempel Logic The logic of science is informal by mathematical logical standards, with no mention of axioms. There are instances of special logical systems, such as quantum logic, that have evolved from certain special cases; however, we do not pursue those systems here. We also do not pursue systems developed on Jaynes work (5). We use a standard proposed by philosophers of science Rudolf Carnap and Carl Hempel. The system was used for normative purposes only and never used to actually describe a system. We now describe the logical system. The Formal Languages for CH The logical system developed primarily by Rudolf Carnap and Carl Hempel, which we refer to as CH, over the period 1936–1977. CH is complex due to nonlogical languages for theory and observation. Each language has a vocabulary V , 2 Copyright  2002 by ASME August 30, 2002 11:19 a structure of objects, functions and relations that are the basis of interpretations. Formally, LL The logical language is a first order predicate calculus with equality. This language may include modality, temporality, etc. LT The theoretical language is the language of the science with no reference to the observational language. This language would be naı̈vely be the normal non-logical language. Generally, there will be no non-logical axioms. Lo0 The pure observational language (Lo) that has only objects and events for its vocabulary. That is, Lo is a pure term language. Lo0 is a logical language that incorporates Lo as terms but introduces quantifiers, temporality, etc. The question of units and statistics, for example, are dealt with here. LM The mixed language LM combines LT and Lo0 in non-vacuous ways. These terms are related to one another through correspondence rules, described in Section Interpretation for CH Interpretation in CH takes place across the four languages. The vocabulary Vo is that of concrete observables: events, objects, and attributes, including issues of units. The relations and properties in Vo must be also observable. Every variable in Lo must take on values of expressions in Lo only. The key issue is that there can be no partial interpretation ofLT based on observations other than those provided by Vo and VT . The mixed language LM introduces a special type of interpretation known as correspondence rules. Correspondence rules are admissible procedures for applying the theory to the observations. They partially interpret VT by specifying observable content. An example of a correspondence rule is the following example: The Resolution Theorem Proving As is well known, one approach to automatic theorem proving is known as resolution theorem proving (6; 7; 9). Let L be a first order predicate language . Let ∆ be a set of statements in L and let Γ be a statement in L representing a conclusion. The resolution principle states that ∆ proves Γ if and only there is no counterexample. Resolution can be implemented in the following manner. Let Σ = ∆[ f:Γg. Σ can be put into conjunctive normal form Σ00 such that the elements of Σ 0 0 are of the form l1; l2; : : : ; lK where each lk is either atomic formula or the negation of an atomic formula. Let Φ and Ψ be two such elements such that φi and psi j are contradictory. If we conjoin the two, then we have a new clause with a term of the form α^:α, an obvious contradiction: Φ[φi Ψ[ψ j Resolution Φ[Ψ 3 Copyright  2002 by ASME August 30, 2002 11:19 We then induce a data base Σ∞ that contains all possible combinations of statements and their derivatives. If we are ever able to generate the empty clause, then we will have derived a contradiction. Why the process works becomes the departure for CH. Consider again Σ00. Let HU be the set of all constants appearing in Σ. HU is called the Herbrand universe. Let HB be the set of all possible terms that can be computed by the functions named in Σ. HB is the Herbrand base and if there is a contradiction then it must come from here because of the interpretation rules. Artificial intelligence has made great use of this mechanism, greatly extending its range to cover applications of diverse logical content through procedural attachment. The difference between applications now is evident: these procedures could well be a super computing application. The CH system considerably complicates this procedure because of there are different languages: L, LT , LM and Lo0 . The relationship of these languages is as follows: L and LT can be mixed in any way, as can Lo and Lo0 but the two groups can only be mixed through LM . The next complication comes from the difference in inference rules between the formal language L on the one hand and LT , LM and Lo0 on the other. The rules for each are discussed in (1). Validation To review, the tree generated by the resolution process represents a proof of the contradiction ∆[:Γ. The tree itself is rooted by the empty clause and the leaves are clauses. Intermediate nodes are the result of the resolution rule Φ[φi Ψ[ψ j Resolution Φ[Ψ where φi and ψ j are contradictory. The tree produced by resolution